Scott Nicholson - Raising reliability of Web search tool research through replication and chaos theory

Because the World Wide Web is a dynamic collection of information, the Web search tools (or "search engines") that index the Web are dynamic. Traditional information retrieval evaluation techniques may not provide reliable results when applied to the Web search tools. This study is the result of ten replications of the classic 1996 Ding and Marchionini Web search tool research. It explores the effects that replication can have on transforming unreliable results from one iteration into replicable and therefore reliable results after multiple iterations. Introduction Web search tools are the only automated retrieval mechanisms available for guidance in the rapidly growing and changing universe of Web documents. In order to better represent this body of literature, the databases underneath these tools are constantly updated. The extremely dynamic nature of Web search tools makes them a very difficult target to study using traditional information retrieval (IR) evaluation methods. In file:///C|/WEBPAGES/bibliomining/nicholson/chaos.html (1 of 15) [8/29/2003 9:27:43 AM] Scott Nicholson Raising reliability of Web search tool research through replication and chaos theory traditional IR evaluations, a set of queries is entered into a search tool, and the results are judged for relevance. Some type of score is generated for each tool and search, results are established, and the project is ended. This model, however, is not be sufficient for Web search tool research. This article replicates a classic Web search tool study to explore this problem and present a solution.